A Sequential and Intensive Weighted Language Modeling Scheme for Multi-Task Learning-Based Natural Language Understanding

Multi-task learning (MTL) approaches are actively used for various natural language processing (NLP) tasks. The Multi-Task Deep Neural Network (MT-DNN) has contributed significantly to improving the performance of understanding (NLU) However, one drawback is that confusion about representation tasks arises during training MT-DNN model. Inspired by internal-transfer weighting MTL in medical imaging, we introduce a Sequential and Intensive Weighted Language Modeling (SIWLM) scheme. SIWLM consists two stages: (1) weighted (SWL), which trains model learn entire sequentially concentrically, (2) (IWL), enables focus on central task. We apply this scheme call MTDNN-SIWLM. Our achieves higher than existing reference algorithms six out eight GLUE benchmark Moreover, our outperforms 0.77 average overall Finally, conducted thorough empirical investigation determine optimal weight each